Redmine Issues with HTML Formatting

Nov 19, 2014

—

I recently had an issue with a client where we had deployed Redmine with an add-on plugin (CKEditor) that displayed all updates to issues as HTML. This resulted in all new issues and content being created with HTML tags but existing/previous content was not and it looked like a big glob on the page. To resolve this, I created a simple script that would connect to the database and update the journals table notes column to add basic paragraph and line breaks to format the output in a clean manner.

I’ll note that I did try the rake task that comes with ckeditor to convert all notes to texttile formatting but it also formatted all of the new content that was already correct so I had to rollback (via database restore).

The script is a simple redmine loop that loops over the results of a SQL query like the following that looks for notes without formatting and longer than 10 characters:

@raw_journal_results = @raw_journal_client.query("
  SELECT
    id
    ,journalized_id
    ,notes
  FROM #{@db_schema}.journals
  WHERE
      notes NOT LIKE '<%' 
    AND 
      LENGTH(notes) > 10
  ").each do |row|
...

Then update the notes to replace all newlines with an HTML break, and add a paragraph markup to the start and end of the notes.

I then set this to run every few minutes as we also use the email fetch feature to pull in email updates and they are not formatted either. This script, when run successfully once, only takes <1 minute to run so it's easy on the system and keeps things formatted properly for new issues.

Comments

8 responses to “Redmine Issues with HTML Formatting”

kandace

October 12, 2017

Do you know how I can keep the easyredmine from stripping html tags when updating a ticket through email? I think this is the code that is causing it. I’ve commented out some of the code and its helpd bring back some of the html but not all of it.

require 'rails-deprecated_sanitizer'

class EasyMailHandler  options[:easy_rake_task],
      :easy_rake_task_info_detail => options[:easy_rake_task_info_detail]
    }
    options = options.deep_dup

    options[:issue] ||= {}

    options[:allow_override] ||= []
    if options[:allow_override].is_a?(String)
      options[:allow_override] = options[:allow_override].split(',')
    end
    options[:allow_override].map! { |s| s.strip.downcase.gsub(/\s+/, '_') }
    # Project needs to be overridable if not specified
    options[:allow_override] < 1
        log_info_msg "#{self.class.name}: ignoring email with X-Loop-Detect bigger than 1"
        return false
      end
    end

    @user = User.having_mail(sender_email).first if sender_email.present?
    if @user && !@user.active?
      case handler_options[:unknown_user]
      when 'accept'
        @user = User.anonymous
      else
        @user = nil
      end
    end

    if @user.nil?
      # Email was submitted by an unknown user
      case handler_options[:unknown_user]
      when 'accept'
        @user = User.anonymous
      when 'create'
        @user = create_user_from_email
        if @user
          log_info_msg "#{self.class.name}: [#{@user.login}] account created"
          add_user_to_group(handler_options[:default_group])
          unless handler_options[:no_account_notice]
            Mailer.account_information(@user, @user.password).deliver
          end
        else
          log_error_msg "#{self.class.name}: could not create account for [#{sender_email}]"
          return false
        end
      else
        @user = User.where(:login => handler_options[:unknown_user]).first unless handler_options[:unknown_user].blank?
        if @user.nil?
          # Default behaviour, emails from unknown users are ignored
          log_info_msg "#{self.class.name}: ignoring email from unknown user [#{sender_email}]"
          return false
        end
      end
    end
    User.current = @user

    obj = dispatch

    if obj.is_a?(ActiveRecord::Base) && (easy_rake_task_info_detail = handler_options[:easy_rake_task_info_detail])
      easy_rake_task_info_detail.entity = obj
      easy_rake_task_info_detail.save
    end

    obj
  end

  def dispatch
    headers = [email.in_reply_to, email.references].flatten.compact
    subject = Redmine::CodesetUtil.replace_invalid_utf8(cleaned_up_subject).to_s
    if headers.detect { |h| h.to_s =~ MESSAGE_ID_RE }
      klass, object_id = $1, $2.to_i
      method_name = "receive_#{klass}_reply"
      if respond_to?(method_name.to_sym, true)
        send method_name, object_id
      else
        # ignoring it
        log_error_msg "#{self.class.name}: cannot find method #{method_name} from email"
        false
      end
    elsif m = subject.match(ISSUE_REPLY_SUBJECT_RE)
      receive_issue_reply(m[1].to_i)
    elsif m = subject.match(MESSAGE_REPLY_SUBJECT_RE)
      receive_message_reply(m[1].to_i)
    else
      dispatch_to_default
    end
  rescue ActiveRecord::RecordInvalid => e
    # TODO: send a email to the user
    log_error_msg "#{self.class.name}: #{e.message}"
    false
  rescue MissingInformation => e
    log_error_msg "#{self.class.name}: missing information from #{user}: #{e.message}"
    false
  rescue UnauthorizedAction => e
    log_error_msg "#{self.class.name}: unauthorized attempt from #{user}"
    false
  end

  def save_email_as_eml(entity)
    filename = Redmine::CodesetUtil.replace_invalid_utf8(cleaned_up_subject).to_s.tr(' ', '_')
    EasyUtils::FileUtils.save_and_attach_email(self.email, entity, filename, User.current)
  end

  def cleaned_up_subject
    super.presence || '(no subject)'
  end

  def stripped_plain_text_body
    return @stripped_plain_text_body unless @stripped_plain_text_body.nil?

    part = self.email.text_part || self.email.html_part || self.email
    body = part.body.decoded
    encoding = pick_encoding(part)
    @stripped_plain_text_body = begin
      convert_to_utf8(body, encoding)
    rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
     # Rails.logger.warning "ENCODING #{encoding} isn't supported"
      #Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
    end

    # strip html tags and remove doctype directive
    #@stripped_plain_text_body = strip_tags(@stripped_plain_text_body.strip)
    #@stripped_plain_text_body.sub! %r{^ attachment.body.decoded,
          :container => obj,
          :author => self.user}
      attachment_or_nothing.files_to_final_location
      attachment_or_nothing.save
      obj.after_new_version_create_journal(attachment_or_nothing)
    else
      obj.attachments.create(
        :file => attachment.body.decoded,
        :filename => attachment.filename,
        :author => self.user,
        :content_type => attachment.mime_type)
    end
  end

  def fix_attached_images_broken_filename
    self.email.all_parts.each do |part|
      if part.mime_type =~ /^image\/([a-z\-]+)$/
        file_extension = $1

        if part.filename.present? && part.filename =~ /\.$/
          # add missing file extension
          part.content_type = part.content_type.gsub(/name=\"?.+\./, "\\0#{file_extension}") if part.content_type
          part.content_disposition = part.content_disposition.gsub(/filename=\"?.+\./, "\\0#{file_extension}") if part.content_disposition #(fixed_disposition)

        elsif part.filename.blank? && part.content_id.present? && part.content_id =~ /([a-z0-9\.]+)@/
          # create missing filename
          part.content_disposition = "inline; filename=\"#{$1}.#{file_extension}\""
        end
      end
    end
  end

  # Destructively extracts the value for +attr+ in +text+
  # Returns nil if no matching keyword found
  def extract_keyword!(text, attr, format=nil)
    keys = [attr.to_s.humanize]
    if attr.is_a?(Symbol)
      keys < '', :locale => self.user.language) if self.user && self.user.language.present?
      keys < '', :locale => Setting.default_language) if Setting.default_language.present?
    end
    keys.reject! { |k| k.blank? }
    keys.collect! { |k| Regexp.escape(k) }
    additional_keys = []
    keys.each do |key|
      key_without_diacritics = key.mb_chars.normalize(:kd).gsub(/[^\x00-\x7F]/n, '').to_s
      additional_keys << key_without_diacritics
      additional_keys << key_without_diacritics.downcase
      additional_keys << key_without_diacritics.upcase
      additional_keys << key.downcase
      additional_keys << key.upcase
    end
    keys.concat(additional_keys)
    keys.uniq!
    format ||= '.+'
    regexp = /^[ ]*(#{keys.join('|')})[ \t]*:[ \t]*(#{format})\s*$/i
    if m = text.match(regexp)
      keyword = m[2].strip
      text.sub!(regexp, '')
    end
    keyword
  end

  def get_keyword(attr, options={})
    @keywords ||= {}
    if @keywords.has_key?(attr)
      @keywords[attr]
    else
      @keywords[attr] = begin
        override = options.key?(:override) ?
          options[:override] :
          (handler_options[:allow_override] & [attr.to_s.downcase.gsub(/\s+/, '_'), 'all']).present?

        if override && (v = extract_keyword!(stripped_plain_text_body, attr, options[:format]))
          v
        elsif !handler_options[:issue][attr].blank?
          handler_options[:issue][attr]
        end
      end
    end
  end

  def plain_text_body
    return @plain_text_body unless @plain_text_body.nil?
    if Setting.text_formatting == 'HTML'
      html_part, text_part, no_mime_part = false, false, false
      if (text_parts = self.email.all_parts.select { |p| p.mime_type == 'text/html' }).present?
        parts = text_parts
        text_part = true
      elsif (html_parts = self.email.all_parts.select { |p| p.mime_type == 'text/html' }).present?
        parts = html_parts
        html_part = true
      else
        parts = [self.email]
        html_part = (Setting.text_formatting == 'HTML')
        no_mime_part = true
      end

      parts.reject! do |part|
        part.attachment?
      end

      @plain_text_body = parts.map do |p|
        body = p.body.decoded
        encoding = pick_encoding(p)
        encoded_body = begin
       #   convert_to_utf8(body, encoding)
        rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
         # Rails.logger.warning "ENCODING #{encoding} isn't supported"
         # Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
        end
        # convert html parts to text
       # p.mime_type == 'text/html' ? self.class.html_body_to_text(encoded_body) : self.class.plain_text_body_to_text(encoded_body)
      end.join("\r\n")

      if Setting.text_formatting == 'HTML'
        if (!html_part && text_part) || (!html_part && !text_part) || no_mime_part
          #@plain_text_body.gsub!(/\n/, '')
        end
      end
    else
      parts = if (text_parts = email.all_parts.select {|p| p.mime_type == 'text/html'}).present?
        text_parts
      elsif (html_parts = email.all_parts.select {|p| p.mime_type == 'text/html'}).present?
        html_parts
      else
        [email]
      end

      parts.reject! do |part|
        part.attachment?
      end

      @plain_text_body = parts.map do |p|
        body = p.body.decoded
        encoding = pick_encoding(p)
        encoded_body = begin
          convert_to_utf8(body, encoding)
        rescue *Redmine::CodesetUtil::ENCODING_EXCEPTIONS
          Rails.logger.warning "ENCODING #{encoding} isn't supported"
          Redmine::CodesetUtil.replace_invalid_utf8(body, encoding)
        end

        # convert html parts to text
       #  p.mime_type == 'text/html' ? self.class.html_body_to_text(encoded_body) : self.class.plain_text_body_to_text(encoded_body)
      end.join("\r\n")
    end
    @plain_text_body
  end

  def cleanup_body(body)
    cleanup_body = body.dup
    #delimiters = Setting.mail_handler_body_delimiters.to_s.split(/[\r\n]+/).reject(&:blank?).map { |s| Regexp.escape(s) }
    #unless delimiters.empty?
   #   regex = Regexp.new("^*(#{ delimiters.join('|') }).*", Regexp::MULTILINE)
    #  cleanup_body = body.gsub(regex, '')
   # end
    #cleanup_body = cleanup_body.strip
    return cleanup_body unless Setting.text_formatting == 'HTML'

   parse_body = Nokogiri::HTML.parse(cleanup_body)
   # ['head', 'meta', 'style', 'script', 'base'].each do |trash|
   #   parse_body.search(trash).remove
   # end
  # remove blank p, that create empty lines.
    parse_body.css('p').each { |p| p.remove if p.content.strip.blank? }

    body_html = parse_body.at('body')

    msg = ''
    msg << (body_html.present? ? body_html.inner_html : parse_body.html)
    msg << ''

    return msg
  end

  protected

  def log_info_msg(err_msg)
    if handler_options[:logger]
      handler_options[:logger].info(err_msg)
    elsif logger
      logger.info(err_msg)
    end
    update_easy_rake_task_info_detail(err_msg)
  end

  def log_error_msg(err_msg)
    if handler_options[:logger]
      handler_options[:logger].error(err_msg)
    elsif logger
      logger.error(err_msg)
    end
    update_easy_rake_task_info_detail(err_msg)
  end

  def update_easy_rake_task_info_detail(err_msg)
    if handler_options[:easy_rake_task_info_detail]
      handler_options[:easy_rake_task_info_detail].update_column(:detail, err_msg)
    end
  end

  def mails_from_and_cc_array(email)
    return [] if email.nil?

    mails = []
    mails.concat(Array.wrap(email.reply_to)) if !email.reply_to.blank?
    mails.concat(Array.wrap(email.from)) if !email.from.blank?
    mails.concat(Array.wrap(email.cc)) if !email.cc.blank?

    mails.flatten.reject(&:blank?).collect { |mail| mail.to_s.strip.downcase }.uniq
  end

  def mails_from_and_cc(email)
    mails_from_and_cc_array(email).join(', ')
  end

  def pick_encoding(part)
    Mail::RubyVer.respond_to?(:pick_encoding) ? Mail::RubyVer.pick_encoding(part.charset).to_s : part.charset
  end

  def convert_to_utf8(str, encoding)
    if !str.nil? && encoding.to_s.downcase == 'utf-7' && Net::IMAP.respond_to?(:decode_utf7)
      str.force_encoding('UTF-8')
      Redmine::CodesetUtil.to_utf8(Net::IMAP.decode_utf7(str), 'UTF-8')
    else
      Redmine::CodesetUtil.to_utf8(str, encoding)
    end
  end

end

Josh

October 17, 2017

Hi Kandace,

Did you manage to get this resolved?

If you didn’t get it resolved – can you confirm that the email being sent does not have a plain text part that is being inserted? Some email clients send both HTML and plain text and render HTML if the client supports it and plain text if not. It looks like the function “stripped_plain_text_body” returns this plain text part if it exists, else it tries to create it from the HTML message format.

Thanks,
Josh
kandace

November 29, 2017

Hey Josh,
Actually no I did not get this resolved. Some of my emails do have plain text like a description that are wrapped in tags. But it seems to be stripping all html properties, elements and inserting its own elements such as class=”msoNormalTable”. I thought that the ckeditor might be doing this or maybe the easyhelpdesk plugin, but I can’t seem to pinpoint it. In all my settings I have it set to html. I would so love your help on solving this!
kandace

November 29, 2017

its updating the journal desc like this in the db:

asdf

From:

esc…
kandace

November 29, 2017

Sorry forgot to use markup

“`

asdf

From: user@hostname.com [mailto:user@hostname.com]

Sent: Wednesday, November 29, 2017 1:08 PM
To: Firstname Lastname
Subject: [Sales > Website Contact Requests] updated – #53557 – Contact Form – FROM: CustomerFirst CustomerLast – DEPARTMENT: Consumer Support (Contact Customer)

“`
kandace

November 29, 2017

Sorry forgot to use markup

“`

asdf

From: user@hostname.com [mailto:user@hostname.com]

Sent: Wednesday, November 29, 2017 1:08 PM
To: Firstname Lastname

“`
Josh

November 29, 2017

Hi Kandace,

Can you send me your redmine install details, plugins and versions, redmine version, etc..? Maybe email them to me vs post here to keep them somewhat private ( josh itsecureadmin com.

Under Redmine -> Administration -> Information

Thanks,
Josh
Sebastian Besler

October 5, 2021

Hy, i have a similar issue with the html tags from ckeditor,
on the search result page, to much html tags shift the usability for reading the text.
and more

Do you know the place for change the Regexp escape,

Thank you, and best regards

Redmine Issues with HTML Formatting

Comments

8 responses to “Redmine Issues with HTML Formatting”

Leave a Reply