Redmine Issues with HTML Formatting

I recently had an issue with a client where we had deployed Redmine with an add-on plugin (CKEditor) that displayed all updates to issues as HTML. This resulted in all new issues and content being created with HTML tags but existing/previous content was not and it looked like a big glob on the page. To resolve this, I created a simple script that would connect to the database and update the journals table notes column to add basic paragraph and line breaks to format the output in a clean manner.

I’ll note that I did try the rake task that comes with ckeditor to convert all notes to texttile formatting but it also formatted all of the new content that was already correct so I had to rollback (via database restore).

The script is a simple redmine loop that loops over the results of a SQL query like the following that looks for notes without formatting and longer than 10 characters:

@raw_journal_results = @raw_journal_client.query("
  SELECT
    id
    ,journalized_id
    ,notes
  FROM #{@db_schema}.journals
  WHERE
      notes NOT LIKE '<%' 
    AND 
      LENGTH(notes) > 10
  ").each do |row|
...

Then update the notes to replace all newlines with an HTML break, and add a paragraph markup to the start and end of the notes.

I then set this to run every few minutes as we also use the email fetch feature to pull in email updates and they are not formatted either. This script, when run successfully once, only takes <1 minute to run so it's easy on the system and keeps things formatted properly for new issues.

7 thoughts on “Redmine Issues with HTML Formatting”

  1. Do you know how I can keep the easyredmine from stripping html tags when updating a ticket through email? I think this is the code that is causing it. I’ve commented out some of the code and its helpd bring back some of the html but not all of it.

    require 'rails-deprecated_sanitizer'
    
    class EasyMailHandler  options[:easy_rake_task],
          :easy_rake_task_info_detail => options[:easy_rake_task_info_detail]
        }
        options = options.deep_dup
    
        options[:issue] ||= {}
    
        options[:allow_override] ||= []
        if options[:allow_override].is_a?(String)
          options[:allow_override] = options[:allow_override].split(',')
        end
        options[:allow_override].map! { |s| s.strip.downcase.gsub(/\s+/, '_') }
        # Project needs to be overridable if not specified
        options[:allow_override] < 1
            log_info_msg "#{self.class.name}: ignoring email with X-Loop-Detect bigger than 1"
            return false
          end
        end
    
        @user = User.having_mail(sender_email).first if sender_email.present?
        if @user && !@user.active?
          case handler_options[:unknown_user]
          when 'accept'
            @user = User.anonymous
          else
            @user = nil
          end
        end
    
        if @user.nil?
          # Email was submitted by an unknown user
          case handler_options[:unknown_user]
          when 'accept'
            @user = User.anonymous
          when 'create'
            @user = create_user_from_email
            if @user
              log_info_msg "#{self.class.name}: [#{@user.login}] account created"
              add_user_to_group(handler_options[:default_group])
              unless handler_options[:no_account_notice]
                Mailer.account_information(@user, @user.password).deliver
              end
            else
              log_error_msg "#{self.class.name}: could not create account for [#{sender_email}]"
              return false
            end
          else
            @user = User.where(:login => handler_options[:unknown_user]).first unless handler_options[:unknown_user].blank?
            if @user.nil?
              # Default behaviour, emails from unknown users are ignored
              log_info_msg "#{self.class.name}: ignoring email from unknown user [#{sender_email}]"
              return false
            end
          end
        end
        User.current = @user
    
        obj = dispatch
    
        if obj.is_a?(ActiveRecord::Base) && (easy_rake_task_info_detail = handler_options[:easy_rake_task_info_detail])
          easy_rake_task_info_detail.entity = obj
          easy_rake_task_info_detail.save
        end
    
        obj
      end
    
      def dispatch
        headers = [email.in_reply_to, email.references].flatten.compact
        subject = Redmine::CodesetUtil.replace_invalid_utf8(cleaned_up_subject).to_s
        if headers.detect { |h| h.to_s =~ MESSAGE_ID_RE }
          klass, object_id = $1, $2.to_i
          method_name = "receive_#{klass}_reply"
          if respond_to?(method_name.to_sym, true)
            send method_name, object_id
          else
            # ignoring it
            log_error_msg "#{self.class.name}: cannot find method #{method_name} from email"
            false
          end
        elsif m = subject.match(ISSUE_REPLY_SUBJECT_RE)
          receive_issue_reply(m[1].to_i)
        elsif m = subject.match(MESSAGE_REPLY_SUBJECT_RE)
          receive_message_reply(m[1].to_i)
        else
          dispatch_to_default
        end
      rescue ActiveRecord::RecordInvalid => e
        # TODO: send a email to the user
        log_error_msg "#{self.class.name}: #{e.message}"
        false
      rescue MissingInformation => e
        log_error_msg "#{self.class.name}: missing information from #{user}: #{e.message}"
        false
      rescue UnauthorizedAction => e
        log_error_msg "#{self.class.name}: unauthorized attempt from #{user}"
        false
      end
    
      def save_email_as_eml(entity)
        filename = Redmine::CodesetUtil.replace_invalid_utf8(cleaned_up_subject).to_s.tr(' ', '_')
        EasyUtils::FileUtils.save_and_attach_email(self.email, entity, filename, User.current)
      end
    
      def cleaned_up_subject
        super.presence || '(no subject)'
      end
    
      def stripped_plain_text_body
        return @stripped_plain_text_body unless @stripped_plain_text_body.nil?
    
        part = self.email.text_part || self.email.html_part || self.email
        body = part.body.decoded
        encoding = pick_encoding(part)
        @stripped_plain_text_body = begin
          convert_to_utf8(body, encoding)
        rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
         # Rails.logger.warning "ENCODING #{encoding} isn't supported"
          #Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
        end
    
        # strip html tags and remove doctype directive
        #@stripped_plain_text_body = strip_tags(@stripped_plain_text_body.strip)
        #@stripped_plain_text_body.sub! %r{^ attachment.body.decoded,
              :container => obj,
              :author => self.user}
          attachment_or_nothing.files_to_final_location
          attachment_or_nothing.save
          obj.after_new_version_create_journal(attachment_or_nothing)
        else
          obj.attachments.create(
            :file => attachment.body.decoded,
            :filename => attachment.filename,
            :author => self.user,
            :content_type => attachment.mime_type)
        end
      end
    
      def fix_attached_images_broken_filename
        self.email.all_parts.each do |part|
          if part.mime_type =~ /^image\/([a-z\-]+)$/
            file_extension = $1
    
            if part.filename.present? && part.filename =~ /\.$/
              # add missing file extension
              part.content_type = part.content_type.gsub(/name=\"?.+\./, "\\0#{file_extension}") if part.content_type
              part.content_disposition = part.content_disposition.gsub(/filename=\"?.+\./, "\\0#{file_extension}") if part.content_disposition #(fixed_disposition)
    
            elsif part.filename.blank? && part.content_id.present? && part.content_id =~ /([a-z0-9\.]+)@/
              # create missing filename
              part.content_disposition = "inline; filename=\"#{$1}.#{file_extension}\""
            end
          end
        end
      end
    
      # Destructively extracts the value for +attr+ in +text+
      # Returns nil if no matching keyword found
      def extract_keyword!(text, attr, format=nil)
        keys = [attr.to_s.humanize]
        if attr.is_a?(Symbol)
          keys < '', :locale => self.user.language) if self.user && self.user.language.present?
          keys < '', :locale => Setting.default_language) if Setting.default_language.present?
        end
        keys.reject! { |k| k.blank? }
        keys.collect! { |k| Regexp.escape(k) }
        additional_keys = []
        keys.each do |key|
          key_without_diacritics = key.mb_chars.normalize(:kd).gsub(/[^\x00-\x7F]/n, '').to_s
          additional_keys << key_without_diacritics
          additional_keys << key_without_diacritics.downcase
          additional_keys << key_without_diacritics.upcase
          additional_keys << key.downcase
          additional_keys << key.upcase
        end
        keys.concat(additional_keys)
        keys.uniq!
        format ||= '.+'
        regexp = /^[ ]*(#{keys.join('|')})[ \t]*:[ \t]*(#{format})\s*$/i
        if m = text.match(regexp)
          keyword = m[2].strip
          text.sub!(regexp, '')
        end
        keyword
      end
    
      def get_keyword(attr, options={})
        @keywords ||= {}
        if @keywords.has_key?(attr)
          @keywords[attr]
        else
          @keywords[attr] = begin
            override = options.key?(:override) ?
              options[:override] :
              (handler_options[:allow_override] & [attr.to_s.downcase.gsub(/\s+/, '_'), 'all']).present?
    
            if override && (v = extract_keyword!(stripped_plain_text_body, attr, options[:format]))
              v
            elsif !handler_options[:issue][attr].blank?
              handler_options[:issue][attr]
            end
          end
        end
      end
    
      def plain_text_body
        return @plain_text_body unless @plain_text_body.nil?
        if Setting.text_formatting == 'HTML'
          html_part, text_part, no_mime_part = false, false, false
          if (text_parts = self.email.all_parts.select { |p| p.mime_type == 'text/html' }).present?
            parts = text_parts
            text_part = true
          elsif (html_parts = self.email.all_parts.select { |p| p.mime_type == 'text/html' }).present?
            parts = html_parts
            html_part = true
          else
            parts = [self.email]
            html_part = (Setting.text_formatting == 'HTML')
            no_mime_part = true
          end
    
          parts.reject! do |part|
            part.attachment?
          end
    
          @plain_text_body = parts.map do |p|
            body = p.body.decoded
            encoding = pick_encoding(p)
            encoded_body = begin
           #   convert_to_utf8(body, encoding)
            rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
             # Rails.logger.warning "ENCODING #{encoding} isn't supported"
             # Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
            end
            # convert html parts to text
           # p.mime_type == 'text/html' ? self.class.html_body_to_text(encoded_body) : self.class.plain_text_body_to_text(encoded_body)
          end.join("\r\n")
    
          if Setting.text_formatting == 'HTML'
            if (!html_part && text_part) || (!html_part && !text_part) || no_mime_part
              #@plain_text_body.gsub!(/\n/, '')
            end
          end
        else
          parts = if (text_parts = email.all_parts.select {|p| p.mime_type == 'text/html'}).present?
            text_parts
          elsif (html_parts = email.all_parts.select {|p| p.mime_type == 'text/html'}).present?
            html_parts
          else
            [email]
          end
    
          parts.reject! do |part|
            part.attachment?
          end
    
          @plain_text_body = parts.map do |p|
            body = p.body.decoded
            encoding = pick_encoding(p)
            encoded_body = begin
              convert_to_utf8(body, encoding)
            rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
              Rails.logger.warning "ENCODING #{encoding} isn't supported"
              Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
            end
    
            # convert html parts to text
           #  p.mime_type == 'text/html' ? self.class.html_body_to_text(encoded_body) : self.class.plain_text_body_to_text(encoded_body)
          end.join("\r\n")
        end
        @plain_text_body
      end
    
      def cleanup_body(body)
        cleanup_body = body.dup
        #delimiters = Setting.mail_handler_body_delimiters.to_s.split(/[\r\n]+/).reject(&:blank?).map { |s| Regexp.escape(s) }
        #unless delimiters.empty?
       #   regex = Regexp.new("^*(#{ delimiters.join('|') }).*", Regexp::MULTILINE)
        #  cleanup_body = body.gsub(regex, '')
       # end
        #cleanup_body = cleanup_body.strip
        return cleanup_body unless Setting.text_formatting == 'HTML'
    
       parse_body = Nokogiri::HTML.parse(cleanup_body)
       # ['head', 'meta', 'style', 'script', 'base'].each do |trash|
       #   parse_body.search(trash).remove
       # end
      # remove blank p, that create empty lines.
        parse_body.css('p').each { |p| p.remove if p.content.strip.blank? }
    
        body_html = parse_body.at('body')
    
        msg = ''
        msg << (body_html.present? ? body_html.inner_html : parse_body.html)
        msg << ''
    
        return msg
      end
    
      protected
    
      def log_info_msg(err_msg)
        if handler_options[:logger]
          handler_options[:logger].info(err_msg)
        elsif logger
          logger.info(err_msg)
        end
        update_easy_rake_task_info_detail(err_msg)
      end
    
      def log_error_msg(err_msg)
        if handler_options[:logger]
          handler_options[:logger].error(err_msg)
        elsif logger
          logger.error(err_msg)
        end
        update_easy_rake_task_info_detail(err_msg)
      end
    
      def update_easy_rake_task_info_detail(err_msg)
        if handler_options[:easy_rake_task_info_detail]
          handler_options[:easy_rake_task_info_detail].update_column(:detail, err_msg)
        end
      end
    
      def mails_from_and_cc_array(email)
        return [] if email.nil?
    
        mails = []
        mails.concat(Array.wrap(email.reply_to)) if !email.reply_to.blank?
        mails.concat(Array.wrap(email.from)) if !email.from.blank?
        mails.concat(Array.wrap(email.cc)) if !email.cc.blank?
    
        mails.flatten.reject(&:blank?).collect { |mail| mail.to_s.strip.downcase }.uniq
      end
    
      def mails_from_and_cc(email)
        mails_from_and_cc_array(email).join(', ')
      end
    
      def pick_encoding(part)
        Mail::RubyVer.respond_to?(:pick_encoding) ? Mail::RubyVer.pick_encoding(part.charset).to_s : part.charset
      end
    
      def convert_to_utf8(str, encoding)
        if !str.nil? && encoding.to_s.downcase == 'utf-7' && Net::IMAP.respond_to?(:decode_utf7)
          str.force_encoding('UTF-8')
          Redmine::CodesetUtil.to_utf8(Net::IMAP.decode_utf7(str), 'UTF-8')
        else
          Redmine::CodesetUtil.to_utf8(str, encoding)
        end
      end
    
    end
    
  2. Hi Kandace,

    Did you manage to get this resolved?

    If you didn’t get it resolved – can you confirm that the email being sent does not have a plain text part that is being inserted? Some email clients send both HTML and plain text and render HTML if the client supports it and plain text if not. It looks like the function “stripped_plain_text_body” returns this plain text part if it exists, else it tries to create it from the HTML message format.

    Thanks,
    Josh

  3. Hey Josh,
    Actually no I did not get this resolved. Some of my emails do have plain text like a description that are wrapped in tags. But it seems to be stripping all html properties, elements and inserting its own elements such as class=”msoNormalTable”. I thought that the ckeditor might be doing this or maybe the easyhelpdesk plugin, but I can’t seem to pinpoint it. In all my settings I have it set to html. I would so love your help on solving this!

  4. Sorry forgot to use markup

    “`

    asdf

    From: user@hostname.com [mailto:user@hostname.com]

    Sent: Wednesday, November 29, 2017 1:08 PM
    To: Firstname Lastname
    Subject: [Sales > Website Contact Requests] updated – #53557 – Contact Form – FROM: CustomerFirst CustomerLast – DEPARTMENT: Consumer Support (Contact Customer)

    “`

  5. Hi Kandace,

    Can you send me your redmine install details, plugins and versions, redmine version, etc..? Maybe email them to me vs post here to keep them somewhat private ( josh itsecureadmin com.

    Under Redmine -> Administration -> Information

    Thanks,
    Josh

Leave a Reply

Your email address will not be published. Required fields are marked *